Skip to content

feat(cala): Phase 7 — full dashboard#141

Merged
daharoni merged 13 commits into
mainfrom
feat/cala-phase-7
Apr 19, 2026
Merged

feat(cala): Phase 7 — full dashboard#141
daharoni merged 13 commits into
mainfrom
feat/cala-phase-7

Conversation

@daharoni
Copy link
Copy Markdown
Contributor

Summary

Phase 7 (design §12) ships the "full dashboard" on top of the Phase 6 "feels alive" baseline. The four-canvas frame panel, traces strip chart, footprints overlay, per-neuron zoom, and NPZ export all land here; real birth/merge events from the extend loop finally flow through the bus instead of Phase 6's metric placeholder.

  • Real structural events (T1-T4): FitPipeline::drain_apply_events + Fitter.drainApplyEvents WASM binding + W2 bus publish → born @(y,x) #id rows in the event feed.
  • 4-canvas frame panel (T5-T7): Preprocessor.processFrameF32WithStages + Fitter.reconstructLastFrame + FrameQuad (raw / hot-pixel / motion / reconstruction).
  • Traces panel (T8-T9): new trace-sample bus event, archive NeuronTraceStore + request-all-traces, uPlot strip chart with click-to-select.
  • Footprints panel (T10-T12): main-thread max-projection accumulator, request-all-footprints (live-id-only via NeuronEventIndex.liveIds), canvas overlay with click hit-test, NeuronZoomPanel showing bbox-cropped footprint + trace sparkline.
  • Export flow (T15): @calab/io writeNpy dtype dispatch + writeNpz (fflate), buildCalaExportNpz producing scipy.sparse-CSC footprints + dense K×T traces, ExportButton in the header.

Exit proven by apps/cala/e2e/phase7-exit.e2e.test.ts driving the real AVI fixture through the full worker pipeline and round-tripping the export NPZ through parseNpz.

Descoped from the original atom list (moved to Phase 8):

  • T13/T14 two-pass mode — the state-transfer plumbing (Footprints across a worker boundary) is substantial, and a paper version without it would be a no-op. Design doc §12 flags it as Phase 8 work.
  • Frame scrubber per stage, events-in-NPZ, Playwright harness, click-a-footprint direct-mutation UX.

Surfaced during live testing, also Phase 8: extend tuning — overlap_fraction_min = 0.3 lets slightly-displaced births slip through the redundancy gate; no SlowBaseline component is seeded, so vignetting / illumination flows into the residual and inflates the variance map. Together they drive the +4 cells per cycle cap-hitting behavior we observed. Belongs on its own fix-branch, not bundled with this phase's UI expansion.

Small fix also in this branch: the dashboard's frame N · epoch M caption now reads the real fit epoch. Phase 6 routed frame-processed through W1, which hardcoded epoch: 0n; run-control now listens to fit's heartbeat for the dashboard counter.

Test plan

  • cargo test -p calab-cala-core passes (new drain_apply_events tests under fitting_apply.rs, WASM builds clean with jsbindings)
  • npm test --workspace cala passes (110 tests incl. new export.test.ts, updated mocks for drainApplyEventsTyped / processFrameF32WithStages / reconstructLastFrame / componentIds / lastTrace)
  • npm run --workspace cala test:e2e passes: phase5 + phase6-extend + phase6-exit + new phase7-exit.e2e.test.ts
  • npm test --workspace @calab/io passes (71 tests, NPZ writer addition)
  • Live browser smoke on .test_data/anchor_v12_prepped.avi: FrameQuad shows 4 stages; TracesPanel populates + click highlights; FootprintsPanel shows outlines on max-proj + click selects; NeuronZoomPanel opens above event feed; Export NPZ downloads and parses in Python

🤖 Generated with Claude Code

daharoni and others added 13 commits April 19, 2026 08:53
…vents (T1)

`drain_apply` kept only aggregate counts — Phase 6 had to surface
extend activity as a `extend.proposed` metric because the per-
mutation identities never left the Rust side. Phase 7 wants real
`birth`/`merge`/`deprecate` events in the event feed, so the fit
pipeline now also exposes `drain_apply_events` which returns one
`AppliedEvent` per successfully-applied mutation:

- Birth: newly-assigned id + class + support/values + weighted-
  centroid patch coords.
- Merge: pair of deprecated ids + new id + class + support/values.
- Deprecate: id + reason.

Stale/invalid rejections are reflected in the `ApplyBatchReport`
but produce no event. `drain_apply` stays on the surface for
callers that only need counts (tests, metrics). Next task wires
this through the WASM binding.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`Fitter.drainApplyEvents(queue)` now exists alongside `drainApply`.
Returns `{ report: [applied, stale, invalid], events: AppliedEvent[] }`
via serde-wasm-bindgen — the event shape mirrors the JS
`PipelineEvent` birth/merge/deprecate variants with `kind` as the
discriminator.

Changes:
- AppliedEvent / DeprecateReason / ComponentClass gain conditional
  serde derives with `rename_all = "camelCase"` so the wire shape
  matches the existing TS union types.
- `packages/cala-core` adapter exports a typed `WasmAppliedEvent`
  union + `drainApplyEventsTyped` wrapper so callers don't repeat
  the `as` cast that wasm-bindgen's `any` return forces.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the Phase 6 `extend.proposed` metric-only emission with a
real `drainApplyEvents` call. Every successfully-applied mutation
now surfaces as a `birth`/`merge`/`deprecate` `PipelineEvent` on
the event bus — archive worker picks them up through the existing
subscriber path, so the event feed finally shows `born @(y,x)`
rows end-to-end.

The `extend.proposed` metric stays (still useful as a flat-line =
quiet-FOV signal), but the JS-side placeholder `mutationToEvent`
that Phase 6 used to fake ids + `patch: [0,0]` is no longer the
source of structural events for the extend path.

Test stubs (`fit.worker.test.ts` + phase5/6 E2Es) pick up the new
`drainApplyEvents` surface + `drainApplyEventsTyped` adapter
export so they continue to cover the fit loop without real WASM.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
On preview-stride frames, W1 now calls the new
`Preprocessor.processFrameF32WithStages` WASM binding and posts
three `frame-preview` messages tagged with `stage`:
'raw' (decoded grayscale), 'hotPixel' (post hot-pixel median),
'motion' (post-motion, what fit sees). The fourth canvas
(`reconstruction`, Ãc) lands in T6 from W2.

Changes:
- New Rust `PreprocessPipeline::process_frame_with_stages` that
  emits the hot-pixel + motion-corrected intermediates alongside
  the final frame. The hot path (`process_frame`) is unchanged —
  the stage-capture cost only pays on preview-stride frames.
- WASM `Preprocessor.processFrameF32WithStages` returning a flat
  `[final || hot || motion]` buffer (3·pixels), sliced on the JS
  side into `subarray` views — no copy on the JS boundary.
- `WorkerOutbound.frame-preview` gains a `stage` field so the
  dashboard wiring can route each stream to its own canvas.
- `run-control` exposes `latestFrames: Accessor<Partial<Record<FrameStage,
  LatestFramePreview>>>`; the legacy `latestFrame` accessor keeps
  pointing at the `motion` stage for the existing SingleFrameViewer.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New WASM binding `Fitter.reconstructLastFrame()` returns `A · c_t`
as a `Float32Array`. Fit worker emits a `frame-preview` with
`stage: 'reconstruction'` at `framePreviewStride` cadence (default
2 to match W1's preview cadence via `run-control`).

`run-control` now also listens to fit's `frame-preview` posts and
routes them into the same `latestFrames` signal that W1 posts land
in, so the 4-canvas panel (T7) can read all four stages uniformly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New `FrameQuad` component composes four `FrameStagePanel` canvases
in a 2×2 grid (design §8 "Frame panel"):

  raw             · hot-pixel
  motion-corrected · reconstruction (Ãc)

Each panel reads a stage from the `latestFrames` signal populated
by W1 (raw / hot-pixel / motion) and W2 (reconstruction). Caption
shows the current frame index + epoch from the dashboard store.

Scrubber is deferred to a later polish task — needs main-thread
frame history per stage, which is a separate data-plumbing change.

`SingleFrameViewer` stays in the tree as a back-compat surface
(reads `latestFrame` = motion stage) but is no longer rendered by
`DashboardLayout`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New bus event `trace-sample` carries `(t, ids, values)` — fit
emits one per vitals-stride frame after the per-metric emissions.
Archive subscribes, routes samples into a new `NeuronTraceStore`
(drop-oldest ring per neuron id), and exposes them through a new
`request-all-traces` query.

Supporting bits:
- New WASM binding `Fitter.componentIds()` so samples carry the
  correct ids even when extend inserts / removes components across
  cycles.
- Trace samples are *excluded* from the event-log ring and the
  neuron-event index so they don't flood the feed or the
  structural-history queries — the traces store is their sole
  sink.
- `archive-client` gains a typed `requestAllTraces(idFilter?)`
  call, routing a new `all-traces` reply with parallel
  `ids[] / times[][] / values[][]` arrays back to callers.

T9 wires the TracesPanel uPlot chart on top.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `uplot` dependency and a `TracesPanel` component that polls
`requestAllTraces` every second, merges per-id time/value arrays
into a single aligned uPlot frame, and renders one stroke per
neuron with a stable hue-hash color. Clicking a trace selects
that neuron via a new shared `selectedNeuronId` signal —
`FootprintsPanel` (T11) and the per-neuron zoom (T12) will read
it to stay in sync.

Dashboard grid gains a `traces` row under the frame panel so
the chart shares the main column with the 4-canvas viewer.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two complementary pieces feeding the upcoming footprints panel:

1. **Main-thread running max projection**. W1 already posts the
   motion-corrected frame as a `frame-preview` for the 4-canvas
   viewer; `run-control` now folds each motion frame into a
   shared `maxProjection` signal (element-wise max over the u8
   preview). Resets when a new run starts. Kept main-thread —
   W1 already owns the data and no archive round-trip is needed.

2. **Archive `request-all-footprints`**. Returns the most recent
   sparse `A`-column snapshot per *live* neuron. Live = latest
   structural event is not a deprecate, via a new
   `NeuronEventIndex.liveIds()` helper. Client gets a typed
   `AllFootprintsReply` with parallel `ids / pixelIndices / values`
   arrays.

T11 wires both into the `FootprintsPanel` overlay.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New component blits the running max projection into a canvas and
strokes each live footprint's 4-connected boundary in its per-id
color. Clicking a boundary pixel (or interior pixel — hit-test
looks up the sparse support) calls `setSelectedNeuronId`; the
selected id gets a thicker white outline so it pops against the
color wall.

Dashboard grid gains a third column: frame quad | footprints |
events, with the traces strip chart spanning the two left columns
underneath.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New `NeuronZoomPanel` opens above the event feed whenever
`selectedNeuronId` is non-null. Polls `requestFootprintHistory` +
`requestAllTraces({idFilter: [id]})` every 2s and renders:

- bbox-cropped grayscale footprint shape (bbox + 2px padding)
- sparkline of the neuron's most recent trace samples (reuses the
  existing vitals `SparkLine` for visual consistency)

Click-through wiring: traces panel → `setSelectedNeuronId`, footprints
panel → `setSelectedNeuronId`, zoom's × button clears it. No modal
overlay — the panel renders inline in the events column so the
event feed just shrinks while a neuron is being inspected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pulls the current footprints + traces from the archive worker and
packs them into a scipy.sparse-CSC-shaped `.npz`:

- `A_data` / `A_indices` / `A_indptr` / `A_shape`: footprint matrix
  as CSC — `scipy.sparse.csc_matrix((data, indices, indptr), shape)`
  loads it directly.
- `footprint_ids`: parallel id vector.
- `C` (K×T), `C_times`, `C_ids`: dense trace matrix padded with NaN
  on the union time axis.
- `height` / `width`: frame geometry.

Events are intentionally omitted from the NPZ for now — JSON-in-zip
is awkward; the structural event log lives in the UI feed only.

Supporting bits:
- `@calab/io` `writeNpy` now dispatches on dtype (Float32 / Uint32 /
  Int32), and a new `writeNpz(arrays)` helper uses `fflate.zipSync`
  as the inverse of the existing `parseNpz`.
- `run-control` keeps the archive worker reference alive after
  natural run completion so export still works in the `stopped`
  state; `stopRun` clears it explicitly.
- New `ExportButton` lives next to the vitals bar. Disabled when
  no archive worker is available; shows an "Exporting…" indicator
  while polling + zipping.

Two-pass (T13) + run-mode toggle (T14) from the original Phase 7
task list were descoped to Phase 8 — full implementation requires
cross-worker `Footprints` state transfer that's non-trivial.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
End-to-end test `apps/cala/e2e/phase7-exit.e2e.test.ts` drives
the real W1/W2/W4 workers against the test AVI and asserts every
Phase 7 wire-protocol deliverable lands intact:

- real `birth` events on the bus via `drainApplyEvents` (T1-T3)
- 3-stage W1 preview streams: raw / hotPixel / motion (T5)
- W2 reconstruction preview frames (T6)
- `request-all-traces` returns per-id trace samples (T8)
- `request-all-footprints` returns live-id sparse-A (T10)
- `buildCalaExportNpz` round-trips through `parseNpz` with the
  expected CSC + K×T shapes (T15)

Stub Fitter now returns real ids from `componentIds`, populates
`lastTrace` with a fixed amplitude, and emits a non-empty
reconstruction frame so the preview-stride path actually posts.

Design doc §12 updated with the Phase 7 exit status, explicit
deferrals (two-pass, scrubber, events-in-NPZ, Playwright, extend
tuning), and the callouts from the live-testing session (loose
redundancy gate + missing SlowBaseline seeding together drive the
+4 cells per cycle behavior on real recordings).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@daharoni daharoni merged commit b0f5bfe into main Apr 19, 2026
7 checks passed
@daharoni daharoni deleted the feat/cala-phase-7 branch April 19, 2026 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant